Fault diagnosis in highly complex and heavily used systems, such as automated teller machines (ATMs), is difficult in the situation where all of the applications executed on the system are provided by the system's vendor due to the diverse and complex nature of the interactions between software and hardware. The complexity of these interactions are increased further when applications run on the system are not provided by the system's vendor but are rather provided by the customer operating the system as the vendor has no control of the syntax and command set used by the customer derived application. This results in the diagnosis of faults with a component device of the system, or within the system's software stack, being extremely difficult to diagnose remotely. In particular, if the fault is associated with the system's software stack it could have occurred at any level from the device firmware to the customer derived application software itself. The diagnosis of such faults is notoriously problematic.
Attempts to diagnose these faults often requires an engineer to try and recreate the fault at a test system. This is labor intensive and is not guaranteed to reproduce the fault as there often too many interacting factors, such as historical device usage, gaps in log entries, etc., to allow the engineer to recreate the fault manually.
Another method used to determine the cause of the fault is the manual examination of log data and instrumentation detail from the failed device and log data of the device's software stack. Such manual examination of log data is labor intensive and, due to the high volume of data to be reviewed and the interaction of parameters, critical events can be overlooked. This can result in the fault being either misdiagnosed, or failing to be diagnosed.